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Abstract 

Numerical  meteorological  models  are  used  to  assist  in  the  prediction  of 
weather  Each  run  of  a  numerical  model  produces  forecasts  of 
meteorological  variables  which  are  used  as  preliminary  predictions  of 
the  future  values  of  these  variables.  These  initial  predictions  are 
referred  to  as  first-guess  values.  Estimation  of  the  mean-square  first- 
guess  error  is  required  in  the  optimal  interpolation  process  in  the 
numerical  prediction  of  atmospheric  variables.  Several  predictors  for 
the  mean-square  error  of  the  first-guess  wind  speeds  are  studied.  The 
results  suggest  that  prediction  using  observed  covariates  tend  to  be 
better  than  those  using  first-guess  covariates.  However,  observed 
covariates  are  not  always  available.  Predictions  using  first-guess 
covariates  are  better  at  the  250  mb  level  than  the  850  or  500  mb  levels. 
Of  those  first-guess  covariates  studied,  first-guess  wind  speed  appears  to 
be  the  best. 


1.     INTRODUCTION  AND  SUMMARY 

Numerical  meteorological  models  are  used  to  assist  in  the  prediction  of 
weather.  Each  run  of  a  numerical  model  produces  forecasts  of  meteorological 
variables  which  are  used  as  preliminary  predictions  of  future  values  of  these 
variables.  These  initial  predictions  are  referred  to  as  first-guess  values.  In  this 
paper  first-guess  values  will  refer  to  the  most  recent  12-hour  forecasts. 

In  certain  areas  of  the  world,  observations  of  forecasted  variables  become 
available.  Prior  to  the  next  run  of  the  numerical  model  a  multivariate 
optimal  interpolation  analysis  updates  a  first-guess  value  of  a  variable  by 


adding  to  it  a  weighted  observed  value  of  the  variable  if  it  is  available.  The 
weight  multiplying  the  observed  value  depends  on  estimates  of  the  mean- 
squared  error  of  the  first-guess  value  and  the  mean-squared  error  of  the 
observation;  cf.  Goerss  et  al.,  [1991,  a,  b].  Thus  it  is  of  importance  to  predict  the 
first-guess  mean-squared  errors. 

The  general  problem  of  modeling  and  predicting  mean-square  errors  is 
important  but  not  widely  studied;  see  Davidian  and  Carroll  (1987),  Nelder  and 
Lee  (1992),  Aitken  (1987),  McCullagh  and  Nelder  (1983). 

In  Jacobs  and  Gaver  (1991,  1992)  statistical  models  for  the  error  of  the  first 
guess  are  used  to  predict  mean-square  error  for  first-guess  wind  components. 
The  models  assume  that  the  error  of  the  first  guess  has  a  normal  distribution 
with  mean  0  and  variance  which  is  a  function  that  is  log-linear  with 
covariates.  Details  of  the  model  are  presented  in  Appendix  A. 

In  this  paper  we  use  data  from  February  1991  to  compare  the  predictive 
ability  of  various  models.  The  data  consist  of  measurements  and  12  hour 
forecasts  (first-guess  values)  of  u  and  v  wind  components  at  the  850  mb,  500 
mb  and  250  mb  pressure  levels  from  93  stations  in  North  America  25N-75N 
for  the  month  of  February  1991.  The  forecasts  are  produced  using  the 
NOGAPS  Spectral  Forecast  Model;  cf.  Hogan  et  al.,  (1991).  Each  station  has 
measurement  and  first-guess  values  for  every  12  hours;  there  are  some 
missing  observations.  These  missing  values  are  deleted  from  the  data  set.  The 
measurement  values  are  subtracted  from  the  first-guess  values  to  obtain 
observations  of  the  error  of  the  first-guess  value. 

Let  U(o;t),  (respectively  V(o;t)),  be  the  observed  w-wind,  (respectively 
p-wind)  component  at  time  t.  Let  U(f;t),  (respectively  V(f;t)),  be  the  first-guess 
w-wind  (respectively  u-wind)  component  at  time  t;  U(f;t),  (respectively  V(f;t)) 


is  the  forecasted  value  of  the  w-wind  (respectively  i>-wind)  component  made 
12  hours  previously.  The  first-guess  error  for  the  w-wind  (respectively  v- 
wind)  component  is 

Yuit)  =  U(/;f)  -  U(o;t);  (respectively  YV(t)  =  V(f;t)  -  V(o;t)).  (1.1) 

The  following  covariates  are  considered  in  the  log-linear  model  for  the  mean- 
square  error  of  the  first  guess. 

(1.2) 


r(o;  t)  =  (U(o;  t)  -  U{o;  t  - 1))2  +  (V(o;  t)  -  V{o;  t  - 1))2 


w(o;t)=  U{o;t)2  +  V(o;t)2 


(1.3) 


r(/;0  =  r(U(/;0-li(/;f-l))2  +  (l/(/;0-V(/;f-l))2' 


(1.4) 


w(f;t)=  U(f;tf  +  V(f;t)2 


(1.5) 


r\t)  = 


\u(f;t)-U(o;t-l)f  +  (V(f;t)-V(o;t-l)f 


(1.6) 


a{o,U;t)  =  \U(o;t)-U(o;t-l)\,    a{o,V;t)  =  \V(o;t)-V(o,t-l)\  (1.7) 


a(f,U;t)  =  \u(f;t)-U(f;t-l)\,    a(f ,V;t)  =  \v(f;t)-V{f ,t-l)\ 


(1.8) 


a(U;t)  =  \U{f;t)-U(o;t-l%    a*(V;t)  =  \v(f;t)-V(o,t-l]  (1.9) 


m(f;t)  =  max(u(f;t),V(f;tj) 


(1.10) 


The  resultant  wind  r(o;t),  (respectively  r(f;t)  and  r*(0),  is  a  measure  of  the 
observed  (respectively  forecasted),  change  in  the  wind.  The  variable  w(o;t), 
(respectively  iv(f;t)),  is  the  observed,  (respectively  forecasted),  wind  speed. 
Higher  wind  speeds  suggest  more  activity  in  the  atmosphere.  The  change  in 
magnitudes  a(o,U;t),  a(f,U;t)  and  a(U;t)  (respectively  a(o,V;t),  a(f,V;t)  and 
a*(V;t)  will  be  used  to  predict  Y\j(t),  (respectively  Yy(0). 

The  data  are  randomly  divided  into  two  sets  called  DA  and  DB. 
Maximum  likelihood  estimates  of  the  parameters  of  the  models  using 
different  covariates  are  computed  using  data  DA  (respectively  DB). 
Nonparametric  models  based  on  binning  are  also  considered.  The  models  are 
then  used  to  predict  the  mean-square  first-guess  errors  in  data  set  DB 
(respectively  DA).  Log-likelihood  functions  and  the  empirical  distribution  of 
the  first-guess  errors  normalized  by  their  predicted  mean-square  errors  are 
used  to  evaluate  the  models'  predictive  ability.  Details  are  given  in  Section  2. 

In  general,  models  which  use  observed  covariates,  e.g.  w(o),  a(o),  have 
more  predictive  ability  than  those  that  use  first-guess  covariates,  e.g.  w(f),  a(f), 
m{f).  The  models  applied  at  the  250  mb  level  appear  to  have  more  predictive 
ability  than  those  for  500  mb  and  850  mb. 

Among  the  one-variate  models  for  the  250  mb  pressure  height,  the 
models  that  statistically  appear  to  have  the  most  predictive  ability  have  as 
their  covariate  w{o),  a(o)  or  r(o).  Those  that  have  less  but  some  predictive 
ability  have  as  their  covariate  a*,  w(f),  m{f)  or  r*.  Finally,  one-variate  models 
using  variates  r(f)  and  a(f)  appear  to  have  little  predictive  ability.  Among 
those  models  for  the  250  mb  pressure  height  that  use  one  first-guess  covariate, 
m(f)  or  w (/)  appear  to  have  the  most  predictive  ability. 


2.     THE  DATA  ANALYSIS 

In  this  section  we  describe  the  data  analysis.  Let  U,-(o;f)  and  L7,(/;0, 
(respectively  V{(o;t)  and  Vj(f;t))  be  the  observed  and  first-guess  w-wind 
(respectively  z>-wind)  component  at  location  i  =  1,  ...,  S  at  time  t.  By  data  we 
mean  the  vector  (U/(o,0,  U»(/,0,  V,(o,0,  V;(/,0,  Ui(o,*-l),  Ui(f,t-1),  V i(o,t-l), 
V,(/,f-l)).  The  data  set  contains  missing  values.  Vectors  containing  these 
missing  values  are  deleted  from  the  data  set.  Once  missing  values  are  deleted, 
there  are  3618  vectors  at  the  850  mb  level,  4100  at  the  500  mb  level,  and  3744  at 
the  250  mb  level.  The  observed  values  are  subtracted  from  the  first-guess 
values  to  obtain  observations  of  the  first-guess  errors  for  each  wind 
component 

Yi(U;t)  =  Ui{f;t)-Ui(o;t) 
Yi(V;t)  =  Vi{f;t)-Vi(o;t). 

The  remaining  data  are  randomly  divided  into  two  sets  called  DA  and  DB 
without  regard  to  the  values  of  the  data,  the  time  /,  or  the  location.  Thus,  data 
from  the  same  location  for  different  times  may  be  in  different  data  sets. 
Models  are  estimated  for  each  pressure  level  using  only  covariates  for  that 
pressure  level.  The  covariates  considered  for  each  wind  component  appear  in 
Appendix  B.  The  general  statistical  model  is  described  in  Appendix  A. 

The  model  is  estimated  using  data  sets  DA,  DB,  and  all  the  data  for  each 
pressure  level.  The  estimated  values  for  the  parameters  for  selected  models 
appear  in  Tables  3A,  4A,  3B,  4B,  3C  and  4C.  Note  that  the  parameter  estimates 
are  usually  positive.  Hence  increased  values  of  the  covariates  are  associated 
with  higher  variance  of  the  first-guess  errors. 

The  models  estimated  from  DA  (respectively  DB)  are  used  to  predict  the 
first-guess  errors  in  data  set  DB  (respectively  DA).  One  measure  used  for 


assessing  a  model's  goodness  of  fit  and  predictive  ability  is  the  value  of  £,  the 
log-likelihood  function  up  to  addition  of  constants  given  in  Appendix  A 
(A. 4);  the  log-likelihood  for  predicting  mean-square  errors  in  DB  using  a 
model  estimated  using  DA  uses  the  first-guess  error  and  covariate(s)  from  DB 
and  the  parameter  estimates  from  DA.  Values  of  £  are  computed  for  data  DA 
(respectively  DB)  using  the  parameters  estimated  using  DB  (respectively  DA); 
these  values  assess  each  model's  predictive  ability.  Values  of  £  are  also 
computed  for  data  DA  (respectively  DB)  using  parameters  estimated  using  DA 
(respectively  DB);  these  values  assess  each  model's  goodness  of  fit. 

Tables  1A,  IB,  1C  present  the  values  of  £  for  one-variate  models  for  the 
different  pressure  levels.  Also  displayed  are  the  values  of  £  for  a  model  in 
which  the  first-guess  errors  are  independent  normally  distributed  with  mean 
0  and  constant  variance  ea. 

Tables  2A,  2B,  2C  present  values  for  £  for  two-variate  models. 

Compare  the  value  of  £c  for  the  model  with  constant  variance  (no 
covariates)  for  DA  (respectively  DB)  fit  using  DA  (respectively  DB)  with  the 
values  of  £  for  DA  (respectively  DB)  using  models  with  parameters  estimated 
using  the  other  half  of  the  data  DB  (respectively  DA).  A  value  of  £  greater 
than  ~£c  indicates  that  the  corresponding  model  fit  with  the  other  half  of  the 
data  describes  the  data  better  than  the  best  constant  variance  model  fit  with 
the  same  data  it  is  used  to  summarize.  For  850  mb  data  those  one-variate 
models  for  which  £>£c  for  DA  and  DB  for  both  wind  components  are  those 
with  variate  r* ,  a  ,  r,  and  w{6).  For  500  mb  the  one-variate  models  are  those 
with  variate  r*,  a(o),  r,  w(o),  and  w{f).  For  250  mb,  the  models  are  those  with 
variate  r*,  a(o),  r,  w(o),  zv(f),  and  m(/). 


To  compare  the  predictive  ability  of  the  models,  the  fraction  of  increase  in 
1,  (^-^c)/pc|  is  computed  where  £c  is  the  maximum  value  of  £  for  the 
constant  variance  model  (with  no  covariates)  estimated  using  data  DA 
(respectively  DB)  compared  to  the  value  of  £  for  DA  using  one-variate 
models  estimated  using  the  other  half  of  the  data  DB  (respectively  DA).  The 
values  of  percentage  of  increase  appear  in  Table  5  for  the  one-variable  models 
with  variate  r* ,  a,  r,  w(o),  w(f),  and  m(f).  Note  that  the  fraction  increase  tends 
to  be  larger  for  the  250  mb  for  the  covariates  using  observed  data,  r,  w(o),  and 
a(o).  The  fraction  also  tends  to  be  larger  for  the  first-guess  covariates  w(f)  and 
m(f)  at  the  250  mb  level. 

Another  measure  of  predictability  is  the  distribution  of  the  first-guess 
errors  divided  by  their  predicted  standard  deviations.  Table  6  displays  the 
moments  of  the  first-guess  errors  of  the  wind  components  DA  (respectively 
DB)  divided  by  the  standard  deviations  that  are  predicted  for  it  using  the 
model  fit  using  data  of  DB  (respectively  DA).  Recall  that  the  models  assume 
that  these  errors  are  normally  distributed  with  mean  0.  Thus,  if  a  model  were 
perfect  then  the  mean  (respectively  standard  deviation,  skewness,  and 
kurtosis)  of  the  normalized  first-guess  errors  would  be  0,  (respectively  1,  0  and 
3).  Of  particular  interest  is  the  kurtosis.  In  this  application,  the  kurtosis  can  be 
thought  of  as  a  measure  of  the  variability  of  the  variance  (cf.  Cramer  page 
356).  Hence,  the  smaller  the  kurtosis,  the  better  the  prediction  of  the  model. 

Table  6  presents  not  only  results  for  the  model  of  Appendix  A  with 
various  covariates  but  also  results  for  a  nonparametric  one-variate  model. 
This  nonparametric  model  is  as  follows.  The  data  in  DA  (respectively  DB)  are 
binned  into  N  bins  according  to  the  value  of  the  ordered  covariate.  For  each 
bin,  the  mean  of  the  square  of  the  wind  speed  errors  corresponding  to  the 


covariates  in  that  bin  is  computed.  To  evaluate  the  predictive  ability  of  the 
model,  the  other  data  set  DB  (respectively  DA)  is  used.  The  predicted  mean- 
square  error  for  a  data  point  in  DB  (respectively  DA)  is  the  mean  of  the  square 
of  the  wind  speed  errors  for  the  bin  determined  from  DA  (respectively  DB) 
the  data  point's  covariate  lies  in. 

Table  6  presents  selected  results  using  data  for  the  250  mb  level.  Results 
for  parametric  models  of  Appendix  A  with  parameters  estimated  by 
maximum  likelihood  (MLE)  and  the  nonparametric  models  with  bins  are 
presented.  Also  displayed  are  the  sample  moments  of  the  first-guess  errors  in 
the  row  labeled  "none".  Displayed  in  the  row  labeled  "constant"  are  the 
sample  moments  for  the  first-guess  errors  divided  by  the  predicted  standard 
deviation  for  a  model  with  constant  variance  ea  fit  using  the  other  half  of  the 
data. 

The  values  of  the  kurtosis  suggest  the  following.  Once  again,  models 
using  the  observed  covariates  a(o)  and  r(o)  appear  to  make  the  best  predictors. 
Among  the  first-guess  covariates,  m(f)  and  w(f)  appear  to  have  comparable 
predictive  ability.  If  a  nonparametric  model  using  a  first-guess  variate  is  being 
considered,  then  using  first-guess  wind  speed  as  the  covariate  seems  to  be  a 
good  choice. 
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APPENDIX  A 
THE  STATISTICAL  MODEL 

In  this  Appendix  we  describe  the  statistical  model.  Let  L/,(o;0  (respectively 
Vi(f;t))  denote  the  observed  u-wind  component  (respectively  first-guess  wind 
component)  at  location  i  at  time  t;  i  =  1,  ...,  S.  Let  V,(o;0,  (respectively  V,(/;0) 
denote  the  observed  p-wind  component  (respectively  first-guess  wind 
component)  at  location  i  at  time  t.  The  first-guess  error  of  the  w-wind 
(respectively  u-wind)  component  at  location  i  at  time  t  is 

Yi(U;t)  =  Ui(o;t)-Ui{f;t) 

(respectively,  (A.l) 

Yi(V;t)  =  Vi(o;t)-V1{f;t)). 

The  model  is  that  [Yi(U;t)t  i  =  1,  ...,  S)  and  {Y,(V;f),  i  =  1,  ...,  S)  are 
independent  random  variables  having  a  normal  distribution  with  mean  0. 
The  variance  of  Yj(U;t)  is  log-linear  with  a  number  of  covariates.  That  is 

Var[Yi(U;t)\Xi(V,t)  =  xi(\) Xi{p;t)  =  Xi{pj\ 


=  exp 


B  +  LPjWXiU''*)'' 


(A.2) 


The  likelihood  function  for   this  model  is   (up  to  multiplication  by 
constants) 


L(a,ft,...,0p) 


«nn«pK 


/     i 


<x  +  LPj{tWj;t) 


f  / 


1     2 


I    V 


a+ZP/(0*i(/;0 
;=1 


(A.3) 


The  log-likelihood  function  is  (up  to  addition  by  constants) 
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i(a.Pl Pp) 


=  II~ 


t       1 


a+%Pj(t)xi{j;t) 


1     2 

^y  ex 


r  ^ 


^+%Pj{t)xi(j;t) 


(A.4) 


The  recursive  procedure  used  to  estimate  the  parameters  (a,  p\,  ...,  ftp)  is 
described  in  Gaver  and  Jacobs  [1991]. 
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APPENDIX  B 
THE  COVARIATES 

In  this  Appendix  we  list  the  covariates  that  were  considered.  As  before  let 
Ui(o;t)  and  l/,(/;0,  (respectively  Vj(o;t)  and  V,-(/;0)  denote  the  observed  and 
first-guess  w-wind  (respectively  i>-wind)  component  at  time  t  for  location  i, 
i  =  1,  ...,  S.  The  covariates  considered  for  the  first-guess  error  of  the  w-wind 
component  are 


Oi(o,U;t 
*Af.U:t 

a*{U;t 
u>i(o;t 

n(o;t 

n(f;t 
n(t 

m(f;t 


=  |U/(o;t)-L7f(o;f-l)| 
=  |LZi(/;r)-U1.(/;r-l)| 
=  |L7I(/;f)-LTI(o;r-l)| 

Ui(o;tf +  Vi(o;t)2] 

[L7I(o;0-^(o;f-l)]2+[VI(o;0-^(/;r-l)]: 

=  [Uf(f;0-U|(f;*-l)f  +  [Vf(o;0-Vi(f;f-l)f] 

=  [U,(f;0-U/(o;*-l)f  +  [V,(f;0-Vi(o;*-l)f] 
=  max(U(/;f)/V(/;r)). 


The  covariates   considered   for   the  first-guess  error  of  the  p-wind 
component  are 
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flI(o/V;0  =  |Vf(o;0-V;(o;f-l)| 

«i(/,V;0  =  |Vj(/;f)-Vf(/;*-l)| 

«i(V;0  =  |v/(/;0-Vf(o;*-l)| 

^i{0'>t)'wi{f>t)'ri{°'t)'ri(f''t)>  and  rf*W- 
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